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1.  Research  Objectives  and  Results 

Introduction 

This  report  describes  the  research  performed  under  AFOSR  Grant  Number 
80-0054,  which  extended  from  January  1,  1980  to  December  31,  1982.  This 
research  addressed  the  problem  of  modeling  discrete-time  communication 
channels  with  memory.  The  objectives  were  to  investigate  a  variety  of  finitistic 
models  and  to  determine  what  channels  they  could  accurately  model.  The 
approach  taken  was  to  develop  appropriate  channel  distance  measures  (i.e., 
quantitative  measures  of  the  degree  to  which  one  channel  or  model  approxi¬ 
mates  another)  and  then  to  characterize  the  class  of  channels  that  can  be 
approximated  with  arbitrary  accuracy  (according  to  the  appropriate  distance 
measure)  by  finitistic  channel  models  of  a  given  type,  e  g.  finite  memory,  finite 
state  or  primitive. 

The  research  was  divided  among  several  areas,  channel  distance  measures, 
channel  approximation  and  representation,  and  extensions  to  continuous  alpha¬ 
bets.  The  results  in  these  areas  are  described  in  the  subsections  below,  follow¬ 
ing  a  brief  summary  of  the  preliminary  work  upon  whic  h  the  research  is  based 
Unless  otherwise  stated  all  channels  are  assumed  to  be  stationary  and  to  have 
finite  alphabets.  The  references  listed  in  brackets  refer  to  the  list  of  publica¬ 
tions  given  in  Section  2. 

Summary  of  preliminary  work 

In  preliminary  work,1  the  d-channel  distance  was  introduced  as  a  measure 
of  the  degree  to  which  one  finite  alphabet  stationary  channel  or  model  approxi¬ 
mates  another.  This  distance  is  a  natural  extension  of  Ornstein's  d-distance  for 

'D.  L.  Neuhoff  and  P.  C.  Shields,  "Channels  with  almost  finite  memory,"  IEEE  Trans.  Inform. 
Th.uory,  vol.  IT-25,  pp.  440-447,  July  1979. 


random  processes.2  Then  the  class  of  channels  that  are  approximate  with  arbi¬ 
trary  accuracy  by  finite  memory  models  was  identified  Such  channels  turned 
out  to  be  characterized  by  the  properties  of  d-continuity  and  conditional 
almost  block  independence  (CAB!)  The  former  is  an  input  memory  decay  pro¬ 
perty  introduced  by  Gray  and  Ornstein3  and  the  latter  is  an  output  memory 
decay  property.  In  addition  it  was  shown  the  class  or  channels  approximate  by 
finite  memory  models  (namely,  the  d-eontinuous  CAB1  channels)  is  identical  to 
the  class  of  charnels  approximate  by  elementary  finite  memory  models  called 
primitive  in  which  the  output  is  simply  a  finite  sliding-block  coding  of  the  input 
joined  with  a  stationary  memoryless  (1.1. D.)  source  of  noise.  Primitive  models, 
therefore,  play  a  fundamental  role. 

Channel  distance  measures 

Ideally,  the  distance  measure  one  uses  in  a  theory  of  channel  approxima¬ 
tion  should  be  a  metric  or  pseudometric  and  should  be  strong  enough  that  chan¬ 
nels  adjudged  to  be  close  have  similar  behavior  and  weak  enough  that  channels 
without  significant  differences  are  adjudged  to  be  close.  More  specifically  one 
would  like  to  use  the  weakest  distance  having  the  property  that  if  two  channels 
are  close  in  this  distance,  then  their  channel  capacities  are  close  and  the 
respective  performances  obtained  using  a  fixed  channel  cods  are  also  close  In 
other  words  capacity  and  code  performance  should  be  continuous  functions  of 
the  channel  with  respect  to  the  chosen  channel  distance  measure.  The  d- 
distance  is  a  pseudometric  and  is  strong  enough  to  have  the  desired  capacity 
and  code  performance  continuity.  Furthermore,  if  two  channels  have  d- 
distance  0  ,  then  they  are  equivalent  in  the  sense  that  for  any  stationary  or 

*D.  S.  Ornstein,  "An  application  of  ergodic  theory  to  probability  theory,"  Ann.  Erob.,  vol.  1,  pp. 
43-65,  1973. 

3R,  M.  Gray  and  h.  S.  Ornstein,  "Block  coding  for  discrete  stationary  d  -conti nuovs  noisy  chan¬ 
nels,"  IEEE  Trans  Inform  Theory,  vol.  IT-25,  pp.  292-306,  May  1979. 


block-stationary  input  process  applied  to  each,  the  respective  input/output  pair 
processes  are  identical.  From  the  communications  point  of  view,  equivalent 
channels  arc  indistinguishable.  On  the  other  hand  the  d -distance  is  so  strong 
that  it  assigns  nonzero  distance  to  some  pairs  of  equivalent  channels.  Hence  it 
v/ould  seem  prudent  to  look  for  weaker  distances. 

In  this  research  four  weaker  distance  measures  were  studied,  which  we 
here  denote  dl,  da,  da.  d4  in  decreasing  order  of  strength.  Each  is  a  pseu- 
dometrie,  all  have  the  desired  capacity  continuity,  all  but  d4  (the  weakest) 
have  the  desired  code  performance  continuity  and  da,  d3  and  d4  assign  zero 
distance  to  equivalent  channels.  Thus  it  would  appear  that  d3  is  the  most 
appropriate  distance  to  use  in  a  channel  approximation  theory.  However,  it  was 
also  discovered  that  for  the  important  class  of  d-continuous  channels 
d  =  d]  =  da  =  d3  and  d  is  uniformly  equivalent  to  d4  .  Thus  for  the  most 
important  case  the  original  d-distancc  has  all  the  desired  properties.  This  is 
fortunate  because  it  is  also  the  easiest  distance  to  work  with.  This  work  is 
described  in  [ *.]. 

Channel  approximation  and  representation 

Channel  entropy  and  primitive  approximation 

in  the  preliminary  work,  the  primitive  models  that  were  demonstrated  to 
approximate  the  d  -continuous,  CAB1  channels  employed  memoryless  noise 
sources  with  continuous  alphabets.  From  a  practical  standpoint  it  is  more  desir¬ 
able  to  have  primitive  models  with  finite  alphabet  noise  Under  this  research 
grant  we  were  aole  to  show  that  the  channels  approximate  by  primitive  chan¬ 
nels  with  infinite  alphabet  noise  are  also  approximate  by  primitive  models  with 
finite  alphabet  noise.  Thus  there  is  no  loss  in  restricting  attention  to  primitive 
channels  with  finite  alphabet  noise  This  raises  the  question  of  just  how  small 
the  noise  alphabet  can  be.  This  is  of  interest  because  the  cost  of  simulating  a 
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channel  with  a  primitive  model  will  in  some  way  depend  on  the  size  of  the  noise 
alphabet.  A  better  question  is  how  much  entropy  must  the  noise  source  have. 
Entropy  can  be  thought  of  as  a  measure  of  the  uncertainty  or  randomness  which 
the  noise  source  possesses  It  was  discovered  that,  to  every  channel  there  is  a 
quantity  Hc  .  dubbed  the  channel  entropy,  such  that  a  given  noise  source  with 
entropy  Hy  can  be  used  to  approximate  (with  arbitrary  accuracy)  any  d- 
continuous,  CADI  channel  whose  entropy  J!c  is  less  than  Hy  .  Conversely,  the 
given  noise  source  cannot  be  used  to  approximate  (with  arbitrary  accuracy)  any 
channel  whose  entropy  Hq  is  larger  than  Hy  .  Roughly  speaking,  channel 
entropy  is  the  maximum  conditional  entropy  of  the  channel  output  given  the 
input,  where  the  maximum  is  over  all  stationary  input  processes.  (It  is  in  some 
sense  a  dual  of  channel  capacity  )  It  follows  that  the  smallest  noise  alphabet  with 
which  to  approximate  a  d-continuous,  CABI  channel  with  a  primitive  model  is 

jq 

the  smallest  integer  no  smaller  than  2  c  .  This  work  is  described  in  [2] 

Nonanticipating  channels 

All  channels  of  practical  interest  are  nonanticipating  (or  equivalently 
causal)  in  the  usual  sense.  The  preliminary  results  on  channel  approximation 
showed  that  any  d-continuous,  CABI  channel,  whether  nonanticipating  or  not, 
could  be  arbitrarily  well  approximated  by  primitive  channels,  which  could  be 
nonanticipating  or  not  In  other  words  the  nonanticipating  nature  of  real  chan¬ 
nels  was  not  taken  into  account.  Under  this  research  grant  it  was  shown  that 
nonanticipating  d-continuous,  CABI  channels  can  be  arbitrarily-well  approxi¬ 
mated  by  nonanticipating  primitive  channels.  This  work  is  described  in  [3], 

Approximation  by  finite  state  models 

Finite  state  models  are  the  most  important  and  widely  used  class  of  chan¬ 
nel  models.  This  is  because  they  require  only  a  finite  number  of  parameters  to 


specify  and  because  it  is  widely  believed  that  they  can  be  used  to  approximate  a 
wide  variety  of  actual  channels.  In  this  grant  the  latter  belief  was  investigated 
by  studying  the  class  of  channels  approximable  by  finite  state  models. 

The  results  obtained  apply  to  indecomposable  finite  state  models,  which  are 
the  important  subclass  of  finite  state  models  in  which  the  effect  of  the  initial 
state  decays  with  time.  For  example,  a  primitive  rrodei  with  a  finite  alphabet 
noise  source  is  finite  state  and  indecomposable.  It  was  found  that  the  class  of 
channels  approximable  by  indecomposable  finite  slate  models  coincides  with  the 
class  approximable  by  nonantieipating  primitive  models.  This  includes  the  class 
of  nonantieipating  d -continuous,  CAB!  channels  plus,  surprisingly,  some  antici¬ 
pating  channels.  Again  this  emphasizes  the  importance  of  primitive  models.  As 
a  partial  converse,  it  was  also  shown  that  if  a  finite  state  model  is  d-continuous 
and  CABI  and  if  its  state  equals  its  output,  then  it  is  indecomposable.  It  would 
also  be  of  interest  to  find  the  class  of  channels  approximable  by  arbitrary  finite 
state  models.  Conceivably,  this  might  be  a  larger  class.  However,  in  the  course 
of  this  investigation  we  encountered  a  deep  unanswered  question  concerning 
Markov  chains  which  needs  to  be  answered  before  the  channel  can  be  answered. 
This  work  is  described  in  L3j. 

Approximation  with  respect  to  the  weaker  channel  distances 

If  any  of  the  distance  measures  weaker  than  d  are  used  in  a  channel 
approximation  theory,  there  is  the  potential  that  more  channels  will  be  approx¬ 
imable  by  primitive,  finite  memory,  or  indecomposable  finite  state  models. 
Although  this  turns  out  to  be  so,  it  was  found  that  there  are  not  many  such  chan¬ 
nels.  In  particular  the  class  of  channels  approximable  by  primitive  models  with 

respect  to  d*  ,  i  =  1 . 4,  equals  the  class  of  d-continuous,  CABI  channels 

plus  all  channels  at  dt  distance  zero  from  this  class.  For  dz  this  class  is 
strictly  larger,  for  d3  this  class  is  larger  still,  and  for  d4  and  d5  this  class  is 
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the  same  as  for  d3  and  consists  of  all  channels  that  are  d-continuous  and  CABI 
or  equivalent  to  such  This  work  is  described  in  [  1  ]- 

The  exact  representation  of  d-  continuous  CABI  channels 

As  previously  described,  and  d-continuous,  CABI  channel  can  be  arbitrarily 
well  ap}iroxim.ated  by  a  primitive  channel,  which  consists  of  a  memoryiess  noise 
source  and  a  finite  sliciing-block  code  that  operates  simultaneously  on  the  chan¬ 
nel  input  and  noise  source  output  Under  this  grant  it  was  also  shown  that  any 
such  channel  can  be  exactly  represented  by  a  model  with  a  memoryless  noise 
source  and  infinite  sliding-block  code. 

Continuous  alphabet  channels 

In  order  to  extend  the  channel  appr  oximation  theory  to  stationary  channels 
with  continuous  alphabets,  a  new  channel  distance  measure,  the  p-distance  was 
introduced.  The  p-distance,  which  is  a  natural  extension  of  the  d  channel  dis¬ 
tance  and  the  p  random  process  distance,4  measures  the  degree  to  which  one 
channel  or  model  approximates  another  relative  to  a  distortion  measure  on  the 
channel  outputs  If  the  distortion  measure  is  chosen  to  be  Hamming  distortion, 
then  the  p-distance  reduces  to  the  d -distance. 

With  respect  to  this  new  channel  distance  one  may  now  seek  to  characterize 
the  ciass  of  channels  approximabie  by  various  types  of  models  such  as  finite 
memory,  primitive  or  finite  state.  Under  this  grant  it  was  found  that  the  class  of 
channels  approximabie  by  finite  memory  or  primitive  models  is  characterized 
by  the  properties  of  weak  p-continuity  and  p-CABl,  where  weak  p-continuity  is 
an  input  memory  decay  condition  that  generalizes  and  slightly  weakens  the  pro¬ 
perty  of  d -continuity  and  where  p-CABl  is  the  natural  generalization  of  the  CAJ31 
property  to  the  case  of  continuous  alphabets  and  distortion  measure  p  .  This 

4R.  H.  Gray,  D.  L.  Neuhof'  and  P.  C.  Shields,  "A  generalization  of  Ornstein's  distance  with  applica¬ 
tions  to  information  theory,”  Ann.  Prob  ,  vol.  ?!I,  pp.  315-328,  April  1975. 


result  shows  that  primitive  channels  again  play  a  fundamental  role,  in  the 
derivation  of  this  result  it  was  found  necessary  to  restrict  attention  to  channels 
and  models  satisfying  an  unobtrusive  finite  "conditional  variance"  condition, 
dubbed  wandering  p~intp.gr able  .  This  condition  is  satisfied  by  virtually  all 
channels  of  practical  interest.  A  number  of  results  concerning  the  p-topology  of 
the  class  of  stationary  channels  were  also  discovered.  This  work  is  reported  in 
[4]. 

2.  Publications  resulting  from  the  grant 

[1]  D.  L.  Neuhoff  and  P.  C  Shields,  "Channel  distances  and  representa¬ 
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[2]  D.  L.  Neuhoff  and  P.  C  Shields.  "Channel  entropy  and  primitive  approx¬ 
imation,”  Ann.  Piab.  vol.  10,  pp.  108-19B,  1982. 

[3]  D.  L.  Neuhoff  and  P.  C.  Shields,  "Indecomposable  finite  state  channels 
and  primitive  approximation,"  IEEE  Trans.  Inform.  Thy.,  vol.  IT-28, 
pp.  1 1-18,  Jan.  1982. 

[4]  R.  Y.  Bick,  "Continuous  alphabet  channels:  the  p-topology,  p-fimte 
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[2]  D.  L.  Neuhoff  and  P.  C.  Shields,  "Finite  state  indecomposable  channels 
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